TITLE OF THE INVENTION 
METHOD OF GENERATING FRAME INTERPOLATION IMAGE AND AN 
APPARATUS THEREFOR 

CROSS-REFERENCE TO RELATED APPLICATIONS 
This application is based upon and claims the 
benefit of priority from prior Japanese Patent 
Application No. 2003-090838, filed March 28, 2003, the 
entire contents of which are incorporated herein by 
reference . 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 

The present invention relates to a method for 
interpolating adjacent flames by at least one 
interpolation frame to shorten a display flame interval 
in reproducing a video. In a hold type image display 
unit which continues displaying a previous flame until 
a new image is written, for example, a liquid crystal 
display and an electro luminescence display, a blurring 
phenomenon by which the eyes of a viewer follow 
movement of a moving object or an unnatural movement 
due to display of an moving image of a few frames is a 
problem in displaying the moving image. 
2. Description of the Related Art 

This problem can be solved by shortening a display 
frame distance. As a method for realizing shortening 
the display frame distance, there is a method of 
generating an interpolation frame using motion 



compensation used in MPEG2 (Motion Picture Experts 
Group phase 2) and interposing the interpolation frame 
between the frames adjacent thereto. In the motion 
compensation in MPEG2 , a motion vector detected by 
block matching method is used. The block matching 
method is a method including dividing a first reference 
frame into a plurality of first blocks, searching a 
second reference frame adjacent to the first reference 
frame for a second block having a maximum correlation 
with respect to each of the first blocks, and computing 
a motion vector from the second block to the first 
block. 

In a frame interpolation method using such a 
motion compensation, when an interpolation frame is 
generated by the motion compensation, at first the 
motion vector between the first reference frame and the 
second reference frame is scaled to a second motion 
vector between the interpolation frame and the first 
reference frame. An interpolation frame is generated 
by carrying out the motion compensation by means of the 
second motion vector obtained by scaling. In other 
words, the end of the second motion vector is fixed on 
the first reference frame. 

Image data of the block of the first reference 
frame that is indicated by the end of the second motion 
vector is copied to the block on the interpolation 
frame that is indicated by the starting point of the 



second motion vector. 

In this method, the starting point position of 
the second motion vector provided by scaling does not 
always coincide with the position of an to-be- 
interpolated block on the interpolation frame. As 
a result, a clearance having no image data or a region 
on which images are superposed is formed on the 
interpolation frame . 

Japanese patent No. 2528103 discloses a frame 
interpolation method without clearance between images 
or superposition of the image. In this prior method, 
the frame interpolation is carried out by computing a 
correlation between forward and backward reference 
frames that are geometrically symmetrical with respect 
to the to-be-interpolated block on the interpolation 
frame. As a result, an interpolation frame can be 
generated directly by a motion vector obtained first 
without scaling a motion vector. In addition, since 
the to-be-interpolated blocks on an interpolation are 
considered as blocks of the same lattice, the inter- 
polation frame does not occur a clearance between 
images and superposition of images. 

On the other hand, Japanese Patent Laid-Open 
No. 2001-24988 discloses a method for a motion 
compensation by computing a reliability of a motion 
compensation on the basis of a differential value 
between images in estimating a motion and a size of 



detected motion vector, and performing a compensation 
by another method when the reliability is low. Since 
the method of patent document 1 determine a motion 
vector only from a correlation between frames forward 
and backward the interpolation frame, the motion vector 
that reflects an actual movement is not always 
provided. For this reason, the quality of the 
interpolation frame deteriorates, resulting in 
deteriorating quality of the whole video. 

On the other hand, the method of a patent document 
2 may theoretically dissolve the problem of the 
deterioration of the interpolation frame due to 
erroneous detection of the motion vector in the patent 
document 1. However, it is difficult for it to be 
realistic to measure reliability of motion compensation 
only by a differential value between images in 
estimating a movement or a size of a motion vector. 
Therefore, it cannot generate always an appropriate 
interpolation frame . 

It is an object of the present invention to 
provide an interpolation image generating method for 
generating an interpolation image with high quality, 
and an apparatus therefor. 

BRIEF SUMMARY OF THE INVENTION 
An aspect of the present invention provides a 
method of generating an interpolated image, comprising: 
inputting a first reference image, a second reference 



image and a third reference image of a video signal 
that continue in terms of time; generating a first 
interpolation image corresponding to a position of the 
second reference image from the first reference image 
and the third reference image; computing a correlation 
value between the second reference image and the first 
interpolation image; generating a second interpolation 
image corresponding to an interpolation position 
between the second reference image and the third 
reference image; and interposing the second inter- 
polation image in the interpolation position when the 
correlation value is more than a threshold. 

Another aspect of the present invention provides 
an apparatus to interpolate a video signal including a 
first reference image, a second reference image and a 
third reference image that continue in terms of time, 
comprising: a first generator to generate a first 
interpolation image put in a position of the second 
reference image, using the first reference image and 
the third reference image; a computation unit 
configured to compute a correlation value between the 
second reference image and the first interpolation 
image; a second generator to generate a second 
interpolation image put in an interpolation position 
between the second reference image and the third 
reference image; and an interpolation unit configured 
to interpolate the second interpolation image in the 
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interpolation position when the correlation value is 
not less than a threshold to generate an interpolated 
video signal. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
5 FIG. 1 is a block diagram of an image display 

system concerning the first embodiment of the present 
invention. 

FIG. 2 is a conceptual diagram of a frame 
interpolation in the first embodiment . 
10 FIG. 3 is a flow chart of a process of a frame 

interpolation in the first embodiment. 

FIG. 4 is a diagram of explaining a transfer 
of frame data between frame memories in the first 
embodiment . 

15 FIG. 5 is a diagram for explaining an example of 

a method of generating the first interpolation frame 
according to the first embodiment. 

FIG. 6 is a diagram for explaining another example 
of a method of generating the first interpolation frame 
20 according to the first embodiment. 

FIG. 7 is a diagram for explaining a still another 
example of a method of generating the first inter- 
polation frame according to the first embodiment. 

FIG. 8 is a diagram showing an example of a frame 
25 interpolation operation of scene change in the first 

embodiment . 

FIG. 9 is a diagram showing an example of another 



frame interpolation operation in scene change in the 

first embodiment. 

FIG. 10 is a block diagram of an image display 
system concerning the second embodiment of the present 
invention . 

FIG. 11 is a diagram for explaining a method of 
generating the second interpolation, frame according to 
the second embodiment. 

FIG. 12 is a flow chart of a process of a frame 
interpolation in the second embodiment. 

FIG. 13 is a block diagram of an image display 
system concerning the third embodiment of the present 
invention . 

FIG. 14 is a conceptual diagram of a frame 
interpolation in the third embodiment. 

FIG. 15 is a flow chart of a process of a frame 
interpolation in the third embodiment. 

FIG. 16 is a block diagram of an image display 
system concerning the fourth embodiment of the present 
invention . 

FIG. 17 is a conceptual diagram of a frame 
interpolation in the fourth embodiment. 

FIG. 18 is a flow chart of a process of a frame 
interpolation in the fourth embodiment. 

FIG. 19 is a flow chart of a process of a frame 
interpolation in the fifth embodiment of the present 
invention. 



FIG. 20 is a flow chart of a process of a frame 
interpolation in the fifth embodiment. 

FIG. 21 is a conceptual diagram of a frame 
interpolation in the sixth embodiment of the present 
invention . 

FIG. 22 is a flow chart of a process of a frame 
interpolation in the sixth embodiment. 

FIG. 23 is a flow chart of a process of a frame 
interpolation in the sixth embodiment. 

FIG. 24 is a block diagram of an image display 
system concerning the seventh embodiment of the present 
invention . 

FIG. 25 is a conceptual diagram of a frame 
interpolation in the seventh embodiment. 

FIG. 26 is a flow chart of a process of a frame 
interpolation in the seventh embodiment. 

FIG. 27 is a conceptual diagram of a frame 
interpolation in the eighth embodiment of the present 
invention . 

FIG. 28 is a flow chart of a process of a frame 
interpolation in the eighth embodiment. 

FIG. 29 is a flow chart of a process of a frame 
interpolation in the eighth embodiment. 

DETAILED DESCRIPTION OF THE INVENTION 

There will now be described the embodiment of the 
present invention referring to the drawing. 
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(First embodiment) 

FIG. 1 shows configuration of a video display 
system related to the first embodiment of the present 
invention. An example that an input image signal 
5 (video signal) 10 is a non-interlace signal 

(progressive signal) of 60 Hz is converted to a non- 
interlace signal of 120 Hz will be described. A frame 
interpolation is realized by generating an inter- 
polation frame on a time center position (interpolation 
10 position) between two images, i.e., two frames adjacent 

to the interpolation frame, and interposing it between 
two reference frames. 

The video signal 10 is input to a frame memory 13, 
the output of frame memory 13 to a frame memory 12, and 
15 the output of the frame memory 12 to a frame memory 11. 

Assuming that pictures held in the frame memories 11, 
12 and 13 are the first, second and third reference 
frames Rl, R2 and R3 . The video signal 10 is input by 
reference frames Rl, R2 and R3 in this order. 
20 A first interpolation frame generator 14 generates 

a first interpolation frame II corresponding to a 
position of the reference frame R2 between the 
reference frames Rl and R3 . An interpolation deter- 
mination unit 15 inspects quality of the interpolation 
25 frame II from the reference frame R2 and interpolation 

frame II and determines whether or not the inter- 
polation is carried out based on an inspection result. 



The second interpolation frame generator 16 generates a 
second interpolation frame 12 corresponding to a time 
center position (interpolation position) between the 
reference frames R2 and R3 when carrying out an 
interpolation according to a determination result of 
interpolation determination unit 15. 

The second interpolation frame 12 is sent to the 
display system 17 along with the reference frames Rl, 
R2 and R3 . In the display system 17, the interpolation 
frame 12 is interposed between the reference frames R2 
and R3 under control of a timing controller 18, and 
a video subjected to the frame interpolation is 
displayed. 

An operation of the present embodiment is 
explained. By interposing an interpolation frame 
between adjacent frames by the frame interpolation, a 
smoother moving image can be played back. However, 
there is no guarantee that an appropriate interpolation 
frame can be always generated. If an interpolation 
frame of low quality is generated and interposed 
between adjacent frames, quality of the whole video 
deteriorates. In the present embodiment, a step to 
inspect quality of the interpolation frame II is 
provided. It is determined according to an inspection 
result whether or not the interpolation carried out by 
interposing the interpolation frame 12 is executed. 
This can evade a problem to deteriorate quality of the 



whole video due to the interpolation frame of low 
quality. 

As shown in FIG. 2 that is a conceptual diagram of 
the frame interpolation of the present embodiment, 
there are executed processes of: (1) generating the 
first interpolation frame II corresponding to a 
position of the second reference frame R2 from the 
first and third reference frames Rl and R3; (2) 
inspecting quality of interpolation frame II with a 
correlation value obtained by comparing the inter- 
polation frame II with the reference frame R2; and 

(3) carrying out an interpolation by obtaining the 
second interpolation frame 12 from the second and third 
reference frames R2 and R3 and interposing it between 
the reference frames R2 and R3, if quality of the 
inspected interpolation frame II is high, that is, if 
the correlation value is not less than a threshold. 

There will be described a concrete process 
procedure in the present embodiment in conjunction with 
a flow chart shown in FIG. 3 hereinafter. 

<Operation of the frame memories 11-13> 
The video signal 10 is input as continuous frame 
data in units of frames input sequentially. As shown 
in FIG. 4, assuming that the frame data that has been 
input at a certain moment is an i frame. It is assumed 
from past accumulation that data of i-3, i-2 and i-1 
frames are stored in the frame memories 11,12 and 13 



respectively (as shown at 0 in FIG. 4) . After i frame 
is input, frame data of the frame memories 11, 12 and 
13 change as follows. 

(1) The frame data in the frame memory 12 is moved 
to the frame memory 11; 

(2) The frame data in frame memory 13 is moved to 
frame memory 12; and 

(3) The input i frame is stored in the frame 
memory 13. 

As thus described, the frame data of i-2, i-1 and 
i frames are stored in the frame memories 11, 12 and 13 
respectively (step S101) . 

In convenience, the frame data of the frame 
memories 11, 12 and 13 are defined as the first, second 
and third reference frames Rl, R2 and R3 . To access 
the frame data of the frame memories 11, 12 and 13 
equalizes to access the reference frame Rl, R2, frame 
data of R3. 

In step S102, the frame (current frame) of a 
currently input video signal is derived from the frame 
memory 13 as the reference frame R3 . A frame before 
one frame from the current frame is derived from the 
frame memory 12 as the reference frame R2 . A frame 
before two frames from the current frame is derived 
from frame memory 11 as the reference frame Rl . The 
first interpolation frame II is generated as follows. 



<Process of the first interpolation frame 
generator 14> 

According to the first interpolation frame 
generator 14, in generating the first interpolation 
frame II, at first a motion vector between the frames 
R2 and R3 is obtained from the first and third 
reference frames Rl and R3, and then the first 
interpolation frame II corresponding to a time position 
of the second reference frame R2 is generated (step 
S103) . An interpolation frame generation method for 
generating the interpolation frame II is not limited in 
particular, but it can be selected from, for example, 
the following various interpolation frame generation 
methods . 

(1) The first interpolation frame generation 
method 

As shown in FIG. 5, the frame Fl is divided into - 
small blocks, the frame F2 is searched for a block with 
a highest correlation with respect to an object one of 
the small blocks to derive a motion vector, and the 
motion vector is scaled to 1/2. The start point of the 
scaled motion vector is fixed at the small block on the 
interpolation frame FI that is on the same position as 
the small block of the frame Fl, and the block on the 
frame F2 that is set by the motion vector is copied to 
a position of the small block on the interpolation 
frame FI . The interpolation frame FI is generated by 
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doing the above serial process to all divided small 
blocks . 

The process is executed by dividing the frame Fl 
into small blocks, so-called forward movement 
estimation. However, it may be executed by dividing 
the frame 2 into blocks, so-called backward motion 
estimation. In addition, the process may be carried 
out by generating the interpolation frame FI by 
selecting a motion vector obtained by the frames Fl and 
F2 and having a highest correlation therebetween, so- 
called bi-directional motion estimation. 

(2) The second interpolation frame generation 
method 

As shown in FIG. 6, the interpolation frame FI is 
divided into small blocks, and a pair of small blocks 
on the frames Fl and F2 that are point-symmetric with 
- respect to an object small block and have a highest 
correlation to derive a motion vector are searched to 
derive a motion vector. An average of a block on the 
frame Fl and a block on the frame F2 which are defined 
by the motion vector is calculated. The average block 
is copied to a position of the small block on the 
interpolation frame. Carrying out the above serial 
process for all divided small blocks generates the 
interpolation frame . 
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(3) The third interpolation frame generation 
method 

The first and second interpolation frame 
generation methods both are a technique of a block 
base, and assume that there is only one motion in a 
block basically. For this reason, when there are a 
plurality of motions, a block distortion may occur in a 
border between the motions. In. contrast, in the third 
interpolation frame generation method, in order to 
prevent such a block distortion from generating is 
selected a method of dividing a block into a high 
correlation region and a low correlation region, 
assigning a motion vector to the high correlation 
region, and searching the low correlation regions 
recursively, as shown in FIG. 7. 

<Process of interpolation determination unit 15> 
Comparing the first interpolation frame II with 
the second reference frame R2 inspects the quality of 
the interpolation frame II. It is determined whether 
or not interpolation is carried out according to an 
inspection result . 

In other words, a correlation value expressing a 
level of a correlation between reference frame R2 and 
interpolation frame 12 is computed (step S104) . If the 
correlation value is not less than a threshold, it is 
determined that the quality of interpolation frame II 
is high. If it is less than the threshold, it is 



determined that the quality is low (step S105) . 
According to a result of this determination, the second 
interpolation frame 12 is generated as described below. 

An evaluation value concerning the quality is set. 
It is determined according to the evaluation value with 
respect to the threshold whether or not the inter- 
polation is to be done. If the evaluation value 
decreases with an increase in the quality, it may be 
determined to do the interpolation when the evaluation 
value is less than the threshold. If the evaluation 
value increases with an increase in the quality, it may 
be determined to do the interpolation when the evalua- 
tion value is not less than the threshold. If the 
evaluation value is considered as the correlation value 
between the reference frame R2 and the interpolation 
frame 12 the former "the evaluation value is less than 
the threshold" and the latter "the evaluation value is 
not more than the threshold" correspond to "the 
correlation value is not less than the threshold". For 
example, the following some values may be used for the 
evaluation value. 

An example based on a frame-to-frame difference of 
the luminance component of a video signal will be 
described hereinafter. The first case is an evaluation 
value referred to as a mean squared error (MSE) , and 
can be expressed as follows. 
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N M 

MSE = — V V <f re f (x, y) - ^omp(^ Y> ) 2 d> 
NM 

x = l y=l 

f re f (x,y) expresses a luminance value in a point 
(x,y) of a reference image (corresponding to the second 
reference frame R2) , f C omp< x 'Y) indicates a luminance 
value in a point (x,y) of an interpolation image 
(corresponding to the interpolation frame II), and 
(N,M) expresses an image size. The above case is an 
example computing for the whole image, but the image 
size may be arbitrarily determined, and a part of 
the image may be an object. An evaluation value of 
equation (1) expresses that the quality increases as it 
decreases . 

The second case is an evaluation value referred to 
as Subjective Mean Squared Error (SMSE) . This is an 
evaluation value considering a characteristic of a 
human eye based on MSE, and expressed as follows. 



SMSE = 



f - N M ^ 

— S Z i d < x ' y) > p 

NM ~ ~ 

x=l y=l ) 



d(x,y) = (a(x,y)g(f a (x,y)-f b (x,y) ) ) (2) 
f re f (x,y) expresses a luminance value in a point 
(x,y) of a reference image (corresponding to the second 
reference frame R2) and f C omp( x 'Y) indicates a 
luminance value in a point (x,y) of an interpolation 
image (corresponding to the interpolation frame 
II) (N,M) expresses an image size, g(x,y) a recognition 
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of an error, and a(x,y) a weight of a point (x,y) . p 
expresses a variable determining a relation between 
importance of a small error and that of a big error. 
For example, the following equation (3) can be used for 
g (x, y) , a (x, y) , and p. 

g(x,y) = M 2 x2 ( f ref ( X 'Y> >" M 2X2 ( f comp< x 'y) ) 
a(x,y) = 1, P = 3 (3) 
M 2 x2 (f(x,y)) expresses a linear average in 
a window of 2*2 size. This describes a lowpass 
characteristic of a human eye. Similarly to MSE of the 
equation (1), the above case is an example computing 
for the whole image, but the image size may be 
arbitrarily determined, and a part of the image may be 
an object. SMSE of the equation (2) represents that 
the quality increases as SMSE decreases similarly to 
MSE of the equation (1) . 

There will be explained an evaluation value 
paid attention to a color-difference signal of an 
image. This can be obtained by expanding MSE of the 
equation (1) and SMSE of the equation (2) . In MSE, 
f (x,y) is assumed a luminance component of a video 
signal. However, this may be substituted for a color 
difference component of the video signal. f(x,y) in 
the equation (1) may be replaced with a color- 
difference signal Cb to be MSE [Cb] . f (x, y) in the 
equation (1) may be replaced with a color-difference 
signal Cr to be MSE [Cr] . Similarly about SMSE, f(x,y) 



in the equation (2) may be replaced with color- 
difference signals Cb and Cr to be SMSE [Cb] and SMSE 
[Cr] . The above condition expresses that the smaller 
all of MSE [Cb], MSE [Cr] , SMSE [Cb] and SMSE [Cr] 
become in value, the higher the quality is. 

It is possible to combine the evaluation value 
based on the above two luminance component and 
evaluation value based on the color difference 
component to use. The evaluation value using all of 
the luminance component and color difference component 
can use the following value. 

MSE [ YCbCr ] = MSE + MSE[Cb] + MSE[Cr] (4) 
SMSE[YCbCr] = SMSE + SMSE[Cb] + SMSE[Cr] (5) 
The equations (4) and (5) are ones obtained by 
expanding MSE and SMSE, and show that quality increase 
as they decrease. 

The pixels agreeing between two frames are counted 
to obtain the number of pixels. The maximum matching 
pel count (MPC) assuming that the pixels that are 
larger in number have high quality is defined as 
another evaluation value. This evaluation value is 
expressed as follows. 



(x,y) of a reference image (corresponding to the 
second reference frame R2) and f C omp( x 'y) indicates a 




f re f (x,y) expresses a luminance value in a point 



luminance value in a point (x, y) of an interpolation 
image (corresponding to the interpolation frame 
II) (N,M) expresses an image size, and T a threshold* 
In here, an example computing for the whole image is 
described. However, the size may be arbitrarily 
determined, and a part of the image may be an object. 
This validation criterion means that the quality 
increases as the evaluation value increases. About 
MPC, MPC[Cb] and MPC[Cr] can be derived with respect to 
the color-difference signal similarly to MSE . f (x, y) 
in the equation (5) may be replaced with the color- 
difference signals Cb and Cr. The evaluation value 
based on the luminance component and that based on the 
color difference component may be combined with each 
other as follows. 

MPC[YCbCr] = MPC + MPC[Cb] + MPC[Cr] (7) 
The following process for matching with another 
evaluation permits to express that the quality 
increases as the evaluation value decreases. 

MPC = NM - MPC ( 8 ) 
The following evaluation value uses a cross- 
correlation function (CCF) . 

M N 

Z Z ^:omp( m ' n)f ref (m, n) 

CCF = : m = 1 n = 1 

M N 

Z Z ^omp( m ' n ) 2 
_m = l n = l 

(9) 



M N 
Z Z f ref(^n)2 

_m = l n = l 



Without using a differential value of luminance, a 
multiplication value of luminance / division value is 
used. This evaluation value shows that a correlation 
increases as it closes to 1.0. However, it is possible 
by carrying out the following process to match the 
evaluation value with another parameter to express that 
the quality increases as the evaluation value 
decreases . 

CCF' = 1.0 - CCF ( 10 ) 
It is possible by replacing f (x,y) in the equation 
(9) with the color-difference signals Cb and Cr to 
obtain CCF [Cb] and CCF [Cr] with respect to the color- 
difference signal similarly to another evaluation 
value. It is possible to combine the evaluation value 
based on the luminance component and the evaluation 
value based on the color difference component to use. 
This can express as follows. 

CCF' [YCbCr] = CCF' + CCF' [Cb] + CCF' [Cr] (11) 
The threshold can be set by, for example, PSNR 
(peak signal to noise ratio) . Acceptability limit in 
PSNR is considered to be 32 [dB] conventionally. An MSE 
value corresponding to the acceptability limit is 
41.0279. Accordingly, if the threshold is set to 
41.0279, it can be assumed that an interpolation is not 
done when it is less than the acceptability limit. 
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<Process of the second interpolation frame 
generator 16> 

According to the above interpolation determina- 
tion, that is, a result of comparing a correlation 
value between the second reference frame R2 and the 
first interpolation frame II with the threshold in step 
S105, when the correlation value is not less than the 
threshold, that is, it is determined that interpolation 
be carried out, the second interpolation frame 12 
corresponding to a time center position between the 
second and third reference frames R2 and R3 is 
generated using a motion vector (step S106) . 

In a case that it is determined that an inter- 
polation is to be carried out, it is possible to 
generate the second interpolation frame 12 using 
various interpolation frame generation methods as shown 
in FIGS. 5, 6 and 7 like generation of the second 
interpolation frame 12. On the other hand, when the 
correlation value is less than the threshold, that is, 
it is determined that the interpolation is not carried 
out, the second reference frame R2 is output as it is 
(step S107) . 

<Process of visual display unit 18> 
In displaying an image on the display unit 17, an 
output timing of a system other than a video display 
system, for example, a speech system is delayed by one 
frame (1/60 sec. in this case) of the video signal 
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under control of the timing controller 18 (step S108) . 
The second reference frame R2 is displayed in sync with 
the display timing of the image (step S109) . The 
second interpolation frame 12 is displayed after 
1/120 second (step S110) . It is assumed that processes 
from generation of the first interpolation frame II to 
generation of the second interpolation frame 12 are 
finished within 1/60 second. 

In the present embodiment as thus described, the 
quality of interpolation frame is inspected. When it 
is determined that quality of interpolation frame is 
low, the interpolation is not carried out. By such a 
manner, it is possible to avoid a problem of degrading 
the whole quality of the moving image that occurs when 
the interpolation frame of low quality is interposed in 
an interpolation position. 

An original frame does not exist in the inter- 
polation position at which an interpolation frame is to 
be interposed. For this reason, the conventional 
manner cannot inspect the quality of interpolation 
frame surely. In contrast, in the present embodiment, 
the quality of the interpolation frame is inspected by 
obtaining an interpolation precision in positions of 
adjacent frames by doing an interpolation with respect 
to the adjacent frames, and predicting an interpolation 
accuracy of an interpolation position based on the 
former interpolation precision. Since the adjacent 



frames and the interpolation frame are near to each 
other in time interval, to predict the inspection 
result in the adjacent frame positions as that in the 
interpolation position is considered to be proper in 
view of a time continuity of the moving image. 

On the other hand, there is a case where a time 
continuity of a moving image cannot be supposed. As 
such an example, for example, occurrence of a scene 
change is presumable. When the scene change occurs, 
the frame before the scene change point has no 
continuity with respect to the following frame. In 
other words, both frames are quite different in image 
from each other. An operation of the present 
embodiment at the time of scene change is described in 
conjunction with FIGS. 8 and 9, hereinafter. 

FIG. 8 shows an operation when an interpolation 
frame is interposed in a scene change part. When the 
first interpolation frame II is generated, since the 
interpolation frame II is formed from an image before 
the scene change and an image after the scene change, 
the generated interpolation frame becomes a frame 
formed of an upper half of the previous frame and a 
lower half of the following frame. Since such an image 
is no good with respect to matching with the second 
reference frame R2, the evaluation value of an 
interpolation determination becomes very bad. Thus, 
the determination that the interpolation is not 
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executed is reached, and the second reference frame R2 
is interposed in the interpolation position as it is. 

As thus described, an inappropriate and unstable 
interpolation frame is not interposed at the time 
of scene change. Therefore, it is found that an 
interpolation determination is done justly. The above 
operation is similar to the case of FIG. 9. In other 
words, it is understood that if continuity of a moving 
image cannot be assumed, an interpolation determination 
function acts normally. 

(Second embodiment) 

FIG. 10 shows an image display system concerning 
the second embodiment of the present invention. The 
second interpolation frame generator 20 comprises a 
vector scaling module 21 which scales a first motion 
vector MV1 input from a first interpolation frame 
generator 14 and a motion vector motion compensation 
module 22 compensating a motion vector using a scaled 
second motion vector MV2 . 

There will be described an embodiment similar to 
the first embodiment wherein an input image signal 
(video signal) is a non-interlaced signal of 60 Hz, an 
interpolation frame is generated in a time center 
position (interpolation frame position) between two 
reference frames adjacent with respect to the non- 
interlaced signal of 60 Hz, and a non-interlaced signal 
of 120 Hz is generated by interposing the interpolation 



frame between the two reference frames. 

Similarly to the first embodiment, in the present 
embodiment, for the purpose of preventing an image 
quality of the whole video from being deteriorated by 
interposing an interpolation frame of low quality, the 
quality of generated interpolation frame is inspected, 
and if it is determined that the quality is low, the 
interpolation is not carried out. The configuration 
that the second interpolation frame generator 15 
generates the second interpolation frame 12 using a 
motion vector MV1 provided with the first interpolation 
frame generator 14 differs from the first embodiment. 
By doing so, a total computation time can be decreased 
in half. 

A process routine of the frame interpolation in 
the present embodiment will be described referring to 
FIGS. 11 and 12. A generation process of the first 
interpolation frame II from steps S201 to S203, an 
interpolation determination process from steps S204 and 
S205, and a display process from steps S208 to S210 in 
FIG. 12 are omitted from explanation because they are 
similar to the first embodiment. Only a generation 
process of the second interpolation frame 12 will be 
described hereinafter. 

<Second interpolation frame generator 20> 
In generating the second interpolation frame, when 
a correlation value between the reference frame R2 and 
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the second interpolation frame II is not less than a 
threshold in step S205, that is, when it is determined 
to carry out an interpolation, at first the first 
motion vector MV1 provided with the first interpolation 
frame generator 14 is scaled according to an inter- 
polation position of the interpolation frame 12 to 
generate a second motion vector MV2, as shown in 
FIG. 11 (Step S206A) . 

The second interpolation frame 12 is generated by 
carrying out a motion compensation using the scaled 
second motion vector MV2 and the second reference 
frames R2 and R3 (step S206B) . On the other hand, when 
the correlation value is less than the threshold, that 
is, it is determined that the interpolation is not done 
in step S205, the second reference frame R2 is output 
as it is (step S207) . The process of the vector 
conversion step S206A and motion compensation step 
S206B will be further concretely explained. 
<Vector scaling step> 

In the vector scaling step S206A, the motion 
vector MV1 provided in generating the first inter- 
polation frame is scaled, that is, the length of the 
vector is converted to make the second motion vector 
MV2 . In this case, for the purpose of interposing the 
second interpolation frame 12 in a time center position 
between the second and third reference frames R2 and 
R3, the length of the first motion vector MV1 is 
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reduced to 1/2 to generate the second motion vector 
MV2. Concretely, assuming that MV1 = (Vx, Vy) , 
MV2 = (sVx, sVy) is calculated by sVx = l/2*Vx, 
sVy = l/2*Vy. 

<Motion compensation step> 

In the motion compensation step S2 06B, the motion 
compensation is done using the block Bl on the third 
reference frame R2 that is indicated by the scaled 
second motion vector MV2 provided in step S2 0 6A to 
generate the interpolation frame 12. The motion 
compensation will be described referring to FIG. 12. 

The interpolation frame II is divided into small 
blocks in generating the first interpolation frame II. 
In this case, the small block similar to the above 
small block is supposed on the interpolation frame 12 
between the second and third reference frames R2 and 
R3. A motion vector corresponding to each small block 
on the interpolation frame 12 is the motion vector MV2 
reduced to 1/2 in length. The block Bl on the third 
reference frame R3, that is indicated by the motion 
vector MV2, is subjected to motion compensation. In 
other words, the interpolation frame 12 is generated by 
copying the block Bl to the small block position on the 
interpolation frame 12. 

According to the present embodiment, a problem of 
degrading quality of the whole video image that occurs 
by interposing an interpolation frame of low quality 
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can be avoided by providing an option of inspecting 
quality of the interpolation frame and not executing 
the interpolation when it is determined that the 
quality of interpolation frame is low, similarly to the 
first embodiment. At the same time, a computation time 
can be reduced to 1/2 by reusing the second motion 
vector obtained in generating the first interpolation 
frame II for generation of the second interpolation 
frame 12. 

(Third embodiment) 

FIG. 13 shows an image display system concerning 
the third embodiment of the present invention. The 
system provides with a vector scaling unit 19 which 
scales the motion vector MV1 output from the first 
interpolation generator 14. The second interpolation 
frame generator 30 comprises an overlap- detection 
module 31, a motion estimation module 32 and a motion 
compensation module 33. 

A frame interpolation process in the present 
embodiment will be described using FIGS. 14 and 15. 
There will be described an embodiment similar to the 
first embodiment wherein an input image signal (video 
signal) is a non-interlaced signal of 60 Hz, an 
interpolation frame is generated in a time center 
position (interpolation frame position) between two 
reference frames adjacent with respect to the non- 
interlaced signal of 60 Hz, and a non-interlaced signal 



of 120 Hz is generated by interposing the interpolation 
frame between the two reference frames. 

In the present embodiment similarly to the first 
embodiment, for the purpose of preventing an image 
quality of the whole video image from being 
deteriorated by interposing an interpolation frame of 
low quality, the quality of generated interpolation 
frame is inspected, and if it is determined that the 
quality is low, the interpolation is not carried out. 
The configuration that the second interpolation frame 
12 is generated using the motion vector MV2 obtained by 
scaling the motion vector MV1 provided with the first 
interpolation frame generator 14 is common with the 
second embodiment. However, the present embodiment 
differs from the second embodiment in a configuration 
using the motion vector MV2 for limiting an actual 
search range without using the motion vector- MV2 as it 
is . 

A process routine of the frame interpolation in 
the present embodiment will be described referring to 
FIGS. 14 and 15. A generation process of the first 
interpolation frame II from steps S301 to S303, an 
interpolation determination process from steps S304 and 
S305, and a display process from steps S308 to S310 in 
FIG. 15 are omitted from explanation because they are 
similar to the first embodiment. Only a generation 
process of the second interpolation frame 12 will be 



- 31 - 



described hereinafter. 

<Process of interpolation frame generator 30> 
In generating the second interpolation frame 12, 
when a correlation value between the reference frame R2 
and the second interpolation frame II is not less than 
a threshold in step S305, that is, it is determined to 
carry out an interpolation, the first motion vector MV1 
provided with the first interpolation frame generator 
14 is scaled according to an interpolation position of 
the interpolation frame 12 to generate a second motion 
vector MV2 , as shown in FIG. 15 (Step S306A) . 

The scaled second motion vector MV2 detects a 
region B2 overlapped on an object block on the inter- 
polation frame 12. The third motion vector MV3 is 
derived by searching for a search region S defined by a 
motion vector passing through the overlap region B2 . 
The interpolation frame 12 is generated by doing a 
motion compensation using the motion vector MV3 and 
the second and third reference frames R2 and R3 . 
(Steps S306B - S306E) . On the other hand, when the 
correlation value is less than the threshold, that is, 
it is determined that the interpolation is not done, in 
step S205, the second reference frame R2 is output as 
it is (step S307) . 

The generation process of the second interpolation 
frame 12 from steps S306A to S306E in the present 
embodiment comprises a vector scaling step S306A, 
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an overlap detection step S306B - S306C, a motion 
estimation step S306D and a motion compensation step 
S306E. The process of steps S306A - S306E will be 
concretely described hereinafter. 
<Vector scaling step> 

The vector scaling step S306A scales the motion 
vector MV1 provided in generating the first inter- 
polation frame, that is, reduces the length of the 
vector MV1 to obtain the second motion vector MV2 . In 
this case, for the purpose of interposing the second 
interpolation frame 12 in a time center position 
between the second and third reference frames R2 and 
R3, the length of the first motion vector MV1 is 
reduced to 1/2 to generate the second motion vector 
MV2. Concretely, assuming that MV1 = (Vx, Vy) , 
MV2 = (sVx, sVy) is calculated by sVx = l/2*Vx, 
■sVy = l/2*Vy. In this case, the scaling is done with 
fixing the end of the vector. 
<Overlap detection step> 

Because the motion vector MV2 is a vector obtained 
by scaling the motion vector MV1 with fixing the end 
thereof, the block concerning the motion vector MV2 
does not coincide with the block grating on the 
interpolation frame 12. Consequently, in the overlap 
search step S306B, it is detected how degree the block 
concerning the motion vector MV2 overlaps on the object 
block on the interpolation frame 12 with respect to 
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what vector. In other words, the object block is moved 
on the second interpolation frame 12 according to the 
motion vector MV2 (step S306B) . Subsequently, an 
actual search region S for the object block on the 
interpolation frame 12 is determined by the third 
motion vector MV3 concerning the overlapped block (step 
S306B) . A search region S is assumed to include, for 
example, four pixels that are top and bottom and right 
and left with respect to the pixel position indicated 
by the motion vector MV3 is assumed. 
<Motion estimation step> 

In motion estimation step S306D, the fourth motion 
vector is estimated by being searched in geometrically 
symmetry with respect to the second and third reference 
frames R2 and R3 with the object block on the inter- 
polation frame 12 being centered. The search region 
used here is the search region S defined by the motion 
vector MV3 detected in the overlap detection steps 
S306B and S306C. 

<Motion compensation step> 

In motion compensation step S306E, a motion 
compensation is carried out using the blocks on the 
second and third reference frames R2 and R3 that are 
defined by a motion vector estimated in the motion 
estimation step S306D to make the second interpolation 
frame 12. The present embodiment provides the same 
effect as the first and second embodiments. Further, 
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it becomes possible to largely decrease a recalculating 
time by defining the actual search region S using 
the second motion vector MV2 found by the second 
interpolation frame generation step. 
(Fourth embodiment) 

FIG. 16 shows an image display system concerning 
the fourth embodiment of the present invention. The 
configuration that the interpolation determination unit 
25 processes in units of blocks is different from the 
above embodiments. 

There will be described an embodiment similar to 
the first embodiment wherein an input image signal 
(video signal) is a non-interlaced signal of 60 Hz, an 
interpolation frame is generated in a time center 
position (interpolation frame position) between two 
reference frames adjacent with respect to the non- 
interlaced signal of 60 Hz, and a non-interlaced signal 
of 120 Hz is generated by interposing the interpolation 
frame between the two reference frames. 

In the present embodiment similarly to the first 
embodiment, for the purpose of preventing an image 
quality of the whole video image from being dete- 
riorated by interposing an interpolation frame of low 
quality, the quality of generated interpolation frame 
is inspected, and if it is determined that the quality 
is low, the interpolation is not carried out. 

The first embodiment basically inspects quality of 
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an interpolation frame using the whole screen. In 
contrast, in the fourth embodiment image degradation 
does not occur in the whole video image even if a part 
of a screen causes an image degradation continuously or 
locally over a plurality of frames. In the present 
embodiment, a region divided every block as shown at © 
in FIG. 17 is inspected to detect continuous or local 
image deterioration. When it is determined by the 
inspection result that only a block causes continuously 
an image deterioration over a plurality of frames, it 
is determined that the interpolation frame is not 
interposed. 

A process routine of the frame interpolation in 
the present embodiment will be described referring to 
FIGS. 17 and 18. A generation process of the first 
interpolation frame II from steps S401 to S403, a 
generation process of the second interpolation frame 12 
in steps S416 and S417, and a display process from 
steps S418 to S420 in FIG. 18 are omitted from 
explanation because they are similar to the above 
embodiments. Only an interpolation process from steps 
S403 to S414 will be described hereinafter. 

As variable saving that an object block was low 
quality continuously over how many frames is prepared 
pool [i]. i is an index number of a block. The pool 
[i] is reset to all zero at the beginning of all 
processes, that is, before starting an interpolation 
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process with inputting a video signal input, 
interpolation determination step> 
In the interpolation determination step, the 
interpolation frame II and the second reference frame 
are inspected every block. It is determined that the 
interpolation is not carried out when a block of low 
quality exists in a plurality of frames in succession. 
The evaluation value for inspection can use various 
methods similar to those described in the first 
embodiment. The block used here may be a block of size 
used in the interpolation frame generation step S403, 
and can select any other size. The interpolation 
determination step will be concretely described. 

The second reference frame R2 is divided into 
15 a plurality of blocks (step S404). This block is a 

block of size used in the first interpolation frame 
generation step S403 or a block of any other size. 

An interpolation flag is prepared as a 
determination flag indicating whether or not an 
20 interpolation is to be done. A true is given as an 

initial value (step S405) . If the interpolation flag 
is true, it is determined that the interpolation is to 
be executed. If it is false, it is determined that the 
interpolation is not to be executed. The process after 
25 the interpolation decision step is a loop process in 

units of a block (step S406) . 

Quality of the interpolation frame II is inspected 
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by comparing the second reference frame R2 in an object 
block position with the first interpolation frame II 
with respect to an i-th block. In other words, a 
correlation value between the reference frame R2 and 
the second interpolation frame II is computed (step 
S407) . Whether or not the correlation value is not 
less than the threshold is examined (step S408) . 

When the correlation value is not less than the 
threshold, that is, the quality is better than the 
threshold, the process advances to step S412 and the 
pool variable is returned to zero (pool [i]=0) . When 
the correlation value is less than the threshold, 
that is, the quality obtained according to the set 
evaluation value is worse than the threshold, the 
process advances to step S409, and 1 is added to the 
pool variable (pool [i] = pool [i]+l). 

Whether or not an i-th block indicates low quality 
over the set number of frames is determined. Assuming 
that the set number of frames is a threshold 2, for 
example. The determination can be performed according 
to whether or not the pool variable is larger than the 
threshold 2. In other words, it is determined whether 
or not pool [i] > threshold 2 (step S410) . If the pool 
[i] > threshold 2, the quality of the block is low 
continuously, so that the interpolation is not 
executed. In other words, A "false" is inserted in the 
interpolation flag (step S411) . Otherwise, since the 
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interpolation is carried out, the interpolation flag is 
maintained to the "true" set in step S405. 

The steps S406 to S412 are carried out repeatedly 
until it is determined that the process is finished for 
all blocks, and then the interpolation determination 
finishes. As thus described, in the present embodi- 
ment, it is possible to realize the frame interpolation 
which can cope with local image degradation by doing 
the interpolation determination every block. 
(Fifth embodiment) 

The fifth embodiment of the present invention will 
be described referring to a flow chart shown in 
FIGS. 19 to 20. An image display system in the present 
embodiment may be similar to a configuration shown in 
FIG. 1 as used in, for example, the first embodiment. 

There will be described an embodiment similarly to 
the above embodiments wherein an input image signal 
(video signal) is a non-interlaced signal of 60 Hz, an 
interpolation frame is generated in a time center 
position (interpolation frame position) between two 
reference frames adjacent with respect to the non- 
interlaced signal of 60 Hz, and a non-interlaced signal 
of 120 Hz is generated by interposing the interpolation 
frame between the two reference frames. 

In the present embodiment similarly to the first 
embodiment, for the purpose of preventing an image 
quality of the whole video image from being 
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deteriorated by interposing an interpolation frame of 
low quality, the quality of generated interpolation 
frame is inspected, and if it is determined that the 
quality is low, the interpolation is not carried out. 

In the fourth embodiment, when a local error 
occurs in terms of time and space, it is determined the 
interpolation is not done to keep quality of the whole 
video image. In this case, if no errors occurred 
before and after an error occurred momentarily, motion 
of a moving image is not smooth for an instant when 
an interpolation is not carried out. Such disorder of 
the motion is very high in visibility, and may give 
uncomfortable feeling to a viewer. Consequently, in 
the present embodiment, there is provided a process to 

compensate a determination result in an interpolation 
position in view of a determination result before and 

after occurrence of an momentary error, whereby the 

uncomfortable feeling due to continuity of time is 

alleviated. 

A process routine of a frame interpolation in the 
present embodiment will be described referring to 
FIGS. 19 and 20. A generation process of the first 
interpolation frame II in step S506, a generation 
process of the second interpolation frame 12 in steps 
S519 and S520, and a display process from steps S521 to 
S523 in FIGS. 19 and 20 are omitted from explanation 
because they are similar to the first embodiments. An 
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interpolation determination process will be described 
hereinafter . 

<Operation of a frame memory> 

An image signal 10 which is a video signal is 
5 input in units of a frame as time continuous frame data 

sequentially. As shown in FIG. 4, assuming that 
the frame data input at a certain moment is i frame. 
In this time, assuming that (i-l)-th, (i-2)-th, 
(i- (L+l) )-th, -(L+2)-th frames are stored frame 

10 memories L+2, L+l, 12, 11 respectively (0-th state 

in L = 0) . When an i-th frame is input, the frame data 
in the frame memories L+2, L+l, 12, 11 are 

transferred as follows: 

(1) Transfer the frame data in the frame memory 12 

15 to the frame memory 11; 

(2) A Transfer the frame data in frame memory 13 
to the frame memory 12; 

(3) Store the input frame data of i frame to the 
frame memory 13. 

20 In this way, the frame data of (i-L+2)-th, 

(i-L+D-th, (i-2)-th, (i-l)-th, i-th frames are stored 
in the frame memories 11,12, L, L+ll, L+2, 

respectively. 

In convenience, the frame data in the frame memory 

25 L is defined as the first reference frame Rl, the data 

in the frame memory L+l is defined as the second 
reference frame R2, and the third data in frame memory 
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L+2 are defined as the third reference frame R3 . To 
access the frame data of the frame memories L, L+l and 
L+2 is to access the frame data of the reference frames 
Rl, R2 and R3 . The above process corresponds to steps 
5 S501 to S507 of FIG. 19. 

interpolation determination step> 
The interpolation determination step compares the 
first interpolation frame 12 generated in step S510 
with the second reference frame R2 to inspect quality 
10 of the interpolation frame II, and determines whether 

or not the interpolation is to be done according to an 
inspection result. In other words, a correlation value 
expressing a level of correlation between the reference 
frame R2 and the interpolation frame 12 is computed 
15 (step S509) . If the correlation value is not less than 

the threshold, quality of the interpolation frame II is 
determined to be high. If it is less than the 
threshold, the quality is determined to be low (step 
S510) . According to the determination result, the 
20 second interpolation frame 12 is generated as follows. 

When the correlation value is determined to be not 
less than the threshold in step S510, "true" is 
interposed in the end of an interpolation list stored 
in, for example, a memory provided in the interpolation 
25 determination unit 15 (step S511). When the correla- 

tion value is determined to be less than the threshold, 
"false" is interposed in the end of the interpolation 
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list (step S512) . Thereafter, the length, of the 
interpolation list is updated from the end to M (step 
S513) . When the median of M is assumed H, it is 
checked whether or not the interpolation list [H] is 
"false" (steps S514 and S515) . If [H] is "false", 
template matching is done with respect to the 
interpolation list [H] . When matching is carried out, 
the interpolation list [H] is set to "true" (step 
S516), and the process advances to step S517. If [H] 
is not "false", the process advances to step S517. 

In step S517, a current frame L is acquired from 
the reference frame 12 as the third reference frame R3 . 
Further, a frame L-l is acquired from the frame memory 
11 as the second reference frame R2 . It is examined 
whether or not the interpolation list [H] is "true" to 
determine whether an interpolation is to be done (step 
S518) . Similarly to the first embodiment, when the 
interpolation is determined, the second interpolation 
frame 12 corresponding to a time center position 
between the second and third reference frames R2 and R3 
is generated (step S520) . When it is determined that 
the interpolation is not done, the second reference 
frame R2 is output as it is (step S519) . 

In performing a display process, the output timing 
of a system other than the image display system is 
delayed by a (L+l) frame of the video signal (step 
S521) . The second reference frame R2 is displayed in 
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sync with a display timing of in image (step S522) . 
Further, the second interpolation frame 12 is displayed 

(step S523) . 

(Sixth embodiment) 
5 The sixth embodiment of the present invention will 

be described using FIGS. 21 to 23. The configuration 
of image display system in the present embodiment is 
similar to that shown in FIG. 1, for example. 

There will be described an embodiment wherein an 
10 input image signal (video signal) is a non-interlaced 

signal of 60 Hz, an interpolation frame is generated in 
a time center position (interpolation frame position) 
between two reference frames adjacent with respect to 
the non-interlaced signal of 60 Hz, and a non- 
15 interlaced signal of 120 Hz is generated by interposing 
the interpolation frame between the two reference 
frames, similarly to the above embodiments. 

A method of generating an interpolation frame may 
select various methods as described in the first 
20 embodiment. However, the various methods have good 

points and bad points. An interpolation frame 
generation method without using a motion vector, 
that is, a method of using a previous frame as an 
interpolation frame or a method of using an average of 
25 forward and backward frames provides a high precision 

for a still region. 

An interpolation frame generation method using a 
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motion vector provides a high precision for a motion 
region. An interpolation frame generation method of a 
region division type may provide a high precision for a 
motion region divided into a plurality of motion parts. 
In the case of this method, it is preferable for 
obtaining a high effect that a region of uniform motion 
is not divided. As described above, an optimum 
interpolation frame generation method differs every 
segment . 

In the present embodiment, an interpolation 
determination step is provided for permitting to select 
an optimum interpolation frame generation method every 
block. As shown at (2) in FIG. 21, the interpolation 
blocks Bll, B12 and B13 generated by a plurality of 
interpolation frame generation methods every block is 
compared with an original block BIO according to an 
interpolation determination. An interpolation frame 
generation method indicating a maximum evaluation value 
is estimated to be optimum even in an actual interpola- 
tion frame position. 

As shown in (3) of FIG. 21, the method is actually 
applied as an interpolation frame generation method 
used for generating an interpolation frame of the 
block . 

The present embodiment employs three interpolation 
frame generation methods described in the first 
embodiment, a method of averaging adjacent forward and 



backward frames and a method of using a previous frame 
as it is. Blocks Bll, B12 and B13 of FIG. 21 show 
interpolation blocks generated by these generation 
methods. An interpolation frame generation method used 
in the present embodiment is not limited to these in 
particular. The method may be selected from, for 
example, three interpolation frame generation methods 
described in the first embodiment. Combination of 
three interpolation frame generation methods, an 
average frame and a previous frame may be applied. 
<First interpolation frame generation step> 
The process including steps S601 and S602 in 
generating the first interpolation frame II is similar 
to the first embodiment. After step S602, a motion 
vector is computed from the first and third reference 
frames Rl and R3, and the interpolation frame II 
corresponding to a time position of the second 
reference frame R2 is generated (step S603) . A method 
of generating the interpolation frame II is not limited 
in particular, and may be selected from the various 
interpolation frame generation methods shown in, for 
example, the first embodiment. For example, image data 
of the first and third reference frames Rl and R3 may 
be subjected to linear average to obtain an average 
frame, and the first reference frame Rl may be adopted 
as a previous frame. 



<Interpolation determination step> 
An optimum interpolation frame generation method 
is selected every in this embodiment. The process of 
an interpolation determination step is a loop process 
done every block (step S606) . With the i-th block (i 
is an index of a block) , the quality of the second 
reference frame R2 and interpolation frame II in an 
interpolation block position, an average frame, and a 
previous frame is inspected (steps S604 to S608) . The 
reference frames Rl and R3 are averaged to generate an 
average frame (step S604) . The reference frame R2 is 
divided into a plurality of small blocks (step S605) . 

In the interpolation block position, a correlation 
value CI between the reference frame R2 and the 
interpolation frame II, a correlation value C2 between 
the reference frame R2 and the average frame generated 
in step S604, and a correlation value C3 between the 
reference frame R2 and the reference frame Rl are 
computed (step S607). The maximum one of these 
correlation values Cl, C2 and C3 is selected (step 
S608) . If C2 is maximum, the average frame is used for 
generating the second interpolation frame 12. If Cl is 
maximum, the interpolation frame II is used. If C3 is 
maximum, the previous frame is used (steps S609, S610, 
S611) . 

The steps S606 to S611 are repeated till it is 
determined in step S612 that they have been done for 
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all the blocks, and the interpolation determination 
finishes . 

<Second interpolation frame generation step> 
An interpolation frame to be interposed between 
the second and third reference frames R2 and R3 is 
generated using an interpolation frame generation 
method selected by the interpolation determination 
step. The process for generating the second inter- 
polation frame is a loop process carried out every 
block (step S613) . With the i-th block (i is an index 
of a block), an interpolation determination result in 
an interpolation block position is read (step S614), 
and the following process is carried out according to 
the result. 

At first, when the interpolation determination 
result is the average frame, an average of the blocks 
of the second and third reference frames is copied to 
an interpolation frame (step S615) . When the inter- 
polation determination result is the interpolation 
frame II, an interpolation block corresponding to a 
time center position between the second and third 
reference frames R2 and R3 in the block position of the 
interpolation frame II is generated using a motion 
vector between the frames R2 and R3, similarly to the 
first embodiment. The interpolation block is copied to 
the interpolation frame (step S616) . Otherwise, an 
interpolation block may be derived by diverting the 
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motion vector of the block position found by the first 
interpolation frame generation step without deriving 
a motion vector newly, as the second embodiment. 
Alternatively, the interpolation block may be derived 
as described in the third embodiment. When the 
interpolation determination result is the previous 
frame, the block in the block position of the second 
reference frame is copied to the interpolation frame 
(step S617) . The above steps are repeated for all 
blocks to generate the interpolation frame 12. In 
addition, when the maximum of the correlation values 
CI, C2 and C3 is compared with the threshold and the 
maximum does not reach the threshold, the second 
reference frame R2 may be used as the interpolation 

15 frame. 

In the present embodiment as described above, 
the interpolation blocks generated by a plurality of 
interpolation frame generation methods are compared 
with an original block every block in the interpolation 
20 determination step. It is estimated that the 

interpolation frame generation method that the 
correlation value becomes maximum is most suitable even 
in an actual interpolation frame position. It is 
adopted as an interpolation frame generation method for 
25 generating the actual interpolation frame. As a 

result, it becomes possible to select an optimum 
interpolation frame generation method every block. 
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(Seventh embodiment) 

There will be described the seventh embodiment 
of the present invention hereinafter. The present 
embodiment permits the precision of quality inspection 
5 with respect to the interpolation frame to improve and 

an high precision interpolation to realize, even if the 
time interval between the frames of an input video 

signal is long. 

FIG. 24 shows a configuration of an image display 
10 system according to the seventh embodiment of the 

present invention. First and second interpolation 
frame generators 14 and 16 are connected in cascade. 
The first interpolation frame generator 14 generates 
the first interpolation frame II from the reference 

15 frames R2 and R3 . 

The second interpolation frame generator 16 
generates the second interpolation frame 12 from the 
first and third interpolation frames II and 12. 
An interpolation determination unit 15 determines 
20 interpolation for inspecting quality of the first 

interpolation frame II from the reference frame R2 and 
the interpolation frame 12. 

There will be described an embodiment wherein an 
input image signal (video signal) is a non-interlaced 
25 signal of 30 Hz, an interpolation frame is generated in 

a time center position (interpolation frame position) 
between two reference frames adjacent with respect to 
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the non-interlaced signal of 30 Hz, and a non- 
interlaced signal of 60 Hz is generated by interposing 
the interpolation frame between the two reference 
frames . 

In the first embodiment, a frame interpolation 
between two reference frame Rl and R3 interposing the 
reference frame Rl therebetween to inspect quality of 
the first interpolation frame II. When the input video 
signal is 30 Hz and a time interval between adjacent 
frames is long (1/30 Hz in this example) as the seventh 
embodiment, the time interval between the reference 
frames Rl and R3 becomes long, too (2/30 Hz in this 
example) . Therefore, if the quality of interpolation 
frame II is inspected by frame interpolation between 
the reference frame" Rl and R3 as the first embodiment, 
the interpolation precision falls down. Alternatively, 
a search region must be extended to maintain the 
interpolation precision. 

Consequently, in the present embodiment, 
interpolation frame for quality inspection is generated 
in frame interpolation of two interpolation frames 
sandwiching one reference frame. Concretely, the 
following processes are done as shown in FIG. 25. 

(1) Interpolation frame II (first interpolation 
frame) corresponding to a time center position between 
the second and third reference frames R2 and R3 is 
generated based the second and third reference frames 
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R2 and R3 . 

(2) Interpolation frame 12 (third interpolation 
frame) corresponding to a time position of the second 
reference frame R2 is generated based on the inter- 
polation frame II and the interpolation frame 10 

(second interpolation frame) which is interposed before 
one frame . 

(3) Quality of the interpolation frame 12 is 
inspected by comparing the second interpolation frame 
12 with the reference frame R2 . 

(4) If quality is higher than a provision, that 
is, a correlation value is not less than the threshold, 
the interpolation frame II is interposed between the 
second and third reference frames R2 and R3 . 

As a result, it is possible to inspect the quality 
of the interpolation frame based on two frames 12 and 
10 between which a time interval is short (in this 
case, the time interval is 1/30 Hz) . Since the inter- 
polation frame 12 is generated by frame interpolation 
between the interpolation frames 12 and 10, quality of 
the interpolation frame 12 is less than good. However, 
the interpolation frame 12 is for inspection first and 
foremost. Therefore, the quality of the interpolation 
frame 12 do not interfere for verification of quality, 
if the evaluation value for quality inspection is 
shifted from the interpolation frame by a value 
required for generating the interpolation frame. 
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Because a frame to be actually interpolated is the 
interpolation frame II generated by the original second 
and third reference frame R2 and R3, the interpolated 
frame II is not contaminated with an error occurring by 
making an interpolation frame from an interpolation 
frame. 

The frame interpolation in the present embodiment 
will be described referring to FIGS. 25 and 26. A 
video signal 10 is input in units of a frame, that is, 
in a form of time continuous frame data sequentially. 
The frame data that is input at a certain moment 
assumes to be i frame. In this time, it is assumed 
that data of i-2 and i-1 frames is stored in the frame 
memories 12 and 13 based on past accumulation (step 
S701) . 

In this state, a currently input video signal 
(current frame) is derived from the frame memory 13 as 
the reference frame R3 . .A frame before one frame than 
the current frame is derived from the frame memory 12 
as the reference frame R2 . An interpolation frame 
before two frames than the current frame is derived 
from the frame memory 11 as the interpolation frame 10 
(step S702) . The first interpolation frame II 
corresponding to a time center position between the 
reference frames R2 and R3 is generated (step S703) . 
The interpolation frame 12 corresponding to a time 
center position of the reference frame R2 is generated 
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from the interpolation frames 10 and II (step S704) . 

A correlation value expressing a level of 
correlation between the interpolation frame 12 and 
the second reference frame R2 is computed to inspect 
quality of the interpolation frame 12 (step S705) . The 
correlation value is compared with the threshold (step 
S706) . If the correlation value is not less than 
threshold, that is, if the quality is more than a 
provision, an interpolation frame II is interposed 
between the second and third reference frames R2 and R3 
(step S707) . If the correlation value does not reach 
the threshold, that is, if the quality of interpolation 
frame II is low, the reference frame R2 is interposed 
between the second and third reference frames R2 and R3 
as it is (step S708) . The display process from steps 
S709 to S713 is omitted in explanation, since it is 
same as the first embodiment. 

In the present embodiment as described above, even 
if the input video signal to be a non-interlace signal 
of 30 Hz, that is, the video signal that a time 
interval between frames is long is subjected to a frame 
interpolation, a high interpolation precision can be 
obtained by inspecting the quality of interpolation 
frame with high precision. 
(Eighth embodiment) 

There will be explained the eight embodiment of 
the present invention. The present embodiment is a 
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combination ob the sixth and seventh embodiment. The 
image display system in the present embodiment is 
similar to the seventh embodiment, and is shown in 
FIG. 24. There will be described an embodiment wherein 
an input image signal (video signal) is a non- 
interlaced signal of 30 Hz, an interpolation frame is 
generated in a time center position (interpolation 
frame position) between two reference frames adjacent 
with respect to the non-interlaced signal of 30 Hz, and 
a non-interlaced signal of 60 Hz is generated by 
interposing the interpolation frame between the two 
reference frames, like the seventh embodiment. 

In the present embodiment, there is described an 
example to select an interpolation frame generation 
method every block similarly to the sixth embodiment. 
In the case of that the input video signal is 30 Hz and 
a time interval between adjacent frames is long as the 
seventh embodiment, when the quality of interpolation 
frame II is inspected by frame interpolation between 
the reference frame Rl and R3, the interpolation 
precision falls down. Alternatively, a search region 
must be extended to maintain the interpolation 
precision. 

In the present embodiment, an interpolation frame 
for inspection is generated without separating the time 
interval from 30 Hz by making the interpolation frame 
12 for quality inspection from the interpolation frames 
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II and 10, like the seventh embodiment- Concretely, 
the interpolation frame is generated as shown in 
FIG. 27. 

(1) Interpolation framed II corresponding to a 
5 time center position between the second and third 

reference frames R2 and R3 are generated by a plurality 
of interpolation frame generation methods. 

(2) Interpolation frames 12 corresponding to a 
time position of the second reference frame R2 are 

10 generated by a plurality of interpolation frame 

generation methods from the interpolation frame II and 
the interpolation frame 10 (interpolation frame 
interposed before one frame) . 

(3) Quality is inspected by comparing the block of 
15 the interpolation frame 12 with an original block 

according to interpolation determination. The 
interpolation frame generation method that the 
evaluation value becomes maximum is estimated to be 
optimum even in an actual interpolation frame position. 

20 (4) The estimated interpolation frame generation 

method is adopted as an interpolation frame generation 
method to be actually used for generating an inter- 
polation frame of the block. In other words, the 
interpolation frame 13 is generated by synthesizing the 

25 interpolation frames II. 

Since the interpolation frame 12 is generated by 
frame interpolation between the interpolation frames 12 
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and 10, quality of the interpolation frame 12 is less 
than good. However, the interpolation frame 12 is for 
inspection first and foremost. Therefore, the quality 
of the interpolation frame 12 do not interfere for 
verification of quality, if the evaluation value for 
quality inspection is shifted from the interpolation 
frame by a value required for generating the inter- 
polation frame. Because a frame to be actually 
interpolated is the interpolation frame II generated by 
the original second and third reference frame R2 and 
R3, the interpolated frame II is not contaminated with 
an error occurring by making an interpolation frame 
from an interpolation frame. 

In the present embodiment, as an interpolation 
frame generation method are employed a method of 
averaging forward and backward adjacent frames, a 
method of using a previous frame as it is, and three 
interpolation frame generation methods described in the 
first embodiment, similarly to the sixth embodiment. 
The interpolation frame generation method is not 
limited to these methods in particular. It may be 
selected from three interpolation frame generation 
methods described in the first embodiment. The three 
interpolation frame generation methods and a method 
using the average frame and the previous frame may be 
adopted. 

FIGS. 29 and 30 show a process routine of a 
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concrete frame interpolation in the present embodiment. 
The process routine of the present embodiment is a 
combination of the process routine of the sixth 
embodiment shown in FIGS. 22 and 23 and the process 
routine of the seventh embodiment shown in FIG. 26. As 
discussed above, according to the present invention, it 
is possible to generate an interpolation frame of high 
quality and carry out frame interpolation. 

Additional advantages and modifications will 
readily occur to those skilled in the art. Therefore, 
the invention in its broader aspects is not limited to 
the specific details and representative embodiments 
shown and described herein. Accordingly, various 
modifications may be made without departing from the 
spirit or scope of the general inventive concept as 
defined by the appended claims and their equivalents. 



